Multi-Speaker Language Modeling

In conventional language modeling, the words from only one speaker are represented at a time, even for conversational tasks such as meetings and telephone calls. In a conversational or meeting setting, however, different speakers can influence each other. In order to recover this missing inter-speaker information, in this work we present a novel approach for conversational language modeling that considers words from other speakers when predicting words from the current one. By adding only one additional word from other speakers into the normal trigram context, our new multi-speaker language model (MSLM) gives a 3.9% perplexity reduction on the Switchboard corpus and a 10.3% perplexity reduction on ICSI Meeting Recorder corpus. This improvement can be further enhanced by the use of class-based multi-speaker language models. We develop two new conditional word clustering algorithms in this framework. With the new algorithms, we achieve a 5.7% perplexity reduction on Switchboard and a 12.2% reduction on the ICSI Meeting Recorder data.